Fix offline peer dependency resolution by synthesizing bun's manifest cache - #103
Draft
typedrat wants to merge 13 commits into
Draft
Fix offline peer dependency resolution by synthesizing bun's manifest cache#103typedrat wants to merge 13 commits into
typedrat wants to merge 13 commits into
Conversation
bun does not keep entry kinds and tuple arities in one-to-one correspondence: github entries can carry an integrity hash (arity 4) and remote or vendored tarball entries can carry inline metadata (arity 3), so dispatching on arity misroutes both — a github dep lands in the npm parser and a pkg.pr.new tarball lands in the git parser, failing with MissingGitRef. Dispatch on the identifier's resolution instead (github:/git+/http(s)://, npm for bare versions at arity 4, file paths otherwise). The identifier is split at the '@' that ends the package name — the second '@' for scoped names — which also fixes resolutions that contain '@' themselves, like https://pkg.pr.new/@scope/pkg@sha tarball URLs. Vendored tarball paths with no file:/./ prefix (e.g. "vendor/pkg-1.0.0.tgz") are now accepted as file packages.
A "<workspace-name>/<pkg>" lockfile entry records its file-dependency path relative to that workspace's directory, but bun.nix paths resolve from the project root — so a vendored tarball under packages/app rendered as ./vendor/pkg.tgz (missing) and a sibling workspace's ../app/vendor/pkg.tgz escaped the root entirely. Factor package building out of convert_lockfile_to_nix_expression into build_packages, and rewrite CopyToStore paths for entries nested under a workspace (longest workspace-name prefix wins) with lexical ././.. normalization.
…network Two hook fixes for failure modes found installing opencode offline. Both stem from the same bun behavior: any drift between package.json and bun.lock makes bun re-resolve the affected dependencies, and re-resolving a git/github/remote-tarball dependency downloads it unconditionally — fatal in the sandbox no matter how complete the cache is. - Detect trustedDependencies / patchedDependencies drift between the root package.json and bun.lock and fail with an actionable message (refresh bun.lock, regenerate bun.nix). Projects routinely commit a lockfile whose copies of these sections lag package.json (opencode's does); without the check the drift surfaces as a wall of ConnectionRefused errors at resolve time with no hint of the cause. - Leave catalog: references pointing at non-registry specs (github:, git+, tarball URLs, file:) unrewritten. bun resolves those natively from the lockfile's catalog section; rewriting them to their resolution registered as a changed spec and forced a re-resolve. The hook now runs the prep script whenever bun.lock exists (previously only when it contained catalog: refs), so the drift check covers every project.
Runs resolve-catalog.ts against three inline fixtures: drifted trustedDependencies/patchedDependencies must fail with a diagnostic naming the drifted entries, identical-but-reordered sections must pass, and catalog: refs must be rewritten to exact versions except when they resolve to non-registry specs. The drifted fixture has no catalog: refs, pinning that the check runs for plain projects too.
…eator Introduce a programs/ cargo workspace with a shared bun2nix-core crate: vendored bun Wyhash11, the bun cache-folder-name port, and verbatim structs + serializer + multi-version builder for bun's binary .npm manifest cache format (bun-npm-manifest-cache-v0.0.7). Rewrite cache-entry-creator from Zig to Rust on top of it, keeping the symlink behavior and adding a manifest mode that turns EntryMeta JSON into .npm cache files. Nix package builds updated to build from the workspace.
The dependency-graph metadata bun stores inline in each npm lockfile entry mirrors the abbreviated registry manifest, so the deserializer rebuilds a per-version VersionMeta from it with no network access. Default-registry entries render a manifest attr in bun.nix (tarball URL, dependency groups, bin/os/cpu, install-script flag) for the offline manifest cache to consume downstream.
Collect EntryMeta records from bun.nix manifest attrs and run cache_entry_creator manifest to write the .npm files bun consults at resolve time, merged into the dependency cache. The install hook exports BUN_MANIFEST_CACHE=2 so bun reads the on-disk manifest cache instead of hitting the network. Manifest-less bun.nix files yield an empty cache and build as before.
…nity#71) react-dom@19 peer-depends on react; bun.lock is excluded from the fixture source, so bun must resolve peer deps from the synthesized manifest cache — the code path that previously fell back to the network and failed in the sandbox. flake.nix now skips auto-importing fixture bun.nix files (they are package expressions, not flake-parts modules). Also pins the manifest format version as a tripwire.
…ity#71) Key each package's .npm manifest by the registry bun will compute at offline-install time. A new bun2nix-core config module parses project-local bunfig.toml/.npmrc contents (.npmrc overrides per key) and ports bun's scope_for_package_name; global/env/CLI layers are deliberately ignored because the Nix sandbox's bun cannot see them either. Non-default entries carry a registry attr inside their bun.nix manifest block, cache-entry-creator groups by (name, registry) and writes <wyhash(name)>-<wyhash(registry)>.npm with the registry's url_hash in the header, and the wasm CLI passes the two config files' contents across the boundary (no filesystem access in core).
…ure) End-to-end guard: the fixture pins react/react-dom to https://registry.npmmirror.com via a committed project-local bunfig.toml; bun.lock is excluded from the source, so sandbox bun must find the registry-keyed <hex>-<hex>.npm manifests under the url_hash it derives from that same config — proving the generation-time key matches byte-for-byte.
…os/cpu Pre-release versions were stored tag-stripped in the releases map, but bun's find_by_version only searches prereleases when the queried version has a tag, so any pre-release pin (e.g. nitro@3.0.1-alpha.1) missed the synthesized manifest and fell back to the network. Parse the full major.minor.patch[-pre][+build] form with a port of bun's Tag.parse state machine, intern pre/build tags (Tag.eql compares wyhash11 of the pre span), and split versions into the releases/prereleases maps, each sorted ascending with bun's numeric-aware order_pre (find_best_version scans from the end). os/cpu parsing now ports bun's Negatable accumulator: "any", "none", "!name" negation, and unrecognized-value handling, all of which bun writes back into bun.lock metadata.
With minimumReleaseAge configured (e.g. opencode's bunfig.toml), bun's manifest cache load rejects any manifest whose has_extended_manifest flag is unset — it demotes the entry to expired and schedules a network refetch, which is fatal in the sandbox. Set the flag: our zeroed publish_timestamp_ms values read as "published at epoch", which passes every age gate — the right answer for lockfile-pinned versions.
typedrat
force-pushed
the
typedrat/offline-manifest-cache
branch
from
July 26, 2026 00:55
372285f to
3489482
Compare
20 tasks
|
I can confirm that this PR works "in the wild" on a monorepo I'm working with at https://github.com/TheRealGramdalf/mindwtr-flake. This fixes a build failure and makes my flake possible, so I'd love to see this merged. Happy to provide more info if needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #71 (and the offline-install portion of #77).
Note
Stacked on #101 and #102 — since PRs from a fork can't target another fork branch, this branch includes their commits; only the last nine commits are new here. I'll rebase and mark it ready once those two land.
bun needs npm package manifests to do peer dependency resolution, even with
--frozen-lockfile. When a dependency graph has peers in it, the lockfile resolution comes back invalid and bun goes looking for the manifest; if it isn't in the cache, it downloads it. Our cache only ever contained the extracted packages and no*.npmmanifest files, so inside the sandbox that download fails withConnectionRefused downloading package manifest. That's why adding a peer-heavy package (the reporter's SvelteKit starter) broke builds of unrelated workspace packages while packages with no peers anywhere kept working: bun never reads a manifest for those.There's no flag to turn this off (
--offlineisn't a real bun flag), and fetching the real registry documents in a fixed-output derivation isn't reproducible since they change whenever a new version is published upstream. So instead we generate bun's binary manifest cache ourselves:bun.lockentry's inline metadata, so bun2nix now reconstructs a per-version manifest from the lockfile and writes it as amanifestattr on the entry inbun.nix. No network involved, which also means it works the same in the wasm CLI.fetchBunDepsfeeds those attrs to a newmanifestsubcommand ofcache-entry-creator, which writes realbun-npm-manifest-cache-v0.0.7files into the cache. The install hook now exportsBUN_MANIFEST_CACHE=2(bun ignores the on-disk cache otherwise), and the manifests getpublic_max_age = u32::MAXso they can't expire mid-build.bun2nix-corecrate now; thesymlinksubcommand keeps the same CLI as the Zig tool and its test vectors came along as Rust tests.Non-default registries work too. bun keys those manifests by a hash of the registry URL from its config, not from the lockfile, so bun2nix parses the project-local
bunfig.toml/.npmrcnext to the lockfile (npmrc wins per key, same as bun) and names the.npmfiles accordingly. Global config, env vars, and--registryare ignored on purpose: the sandboxed bun can't see any of them, so a registry configured only there could never resolve offline anyway, and reading them would bake machine-specific state (and auth tokens) intobun.nix.Pre-release versions work too. bun looks a version up in the manifest's
releasesorprereleasesmap depending on whether it has a tag, and tag equality compares a hash of the pre-release span, so3.0.1-alpha.1has to be stored in the right map with the right hash — an earlier revision stripped the tag and stored it as3.0.1, which bun can never find. The version parser now follows bun'sTag.parsestate machine, and the os/cpu fields go through a port of bun'sNegatablehandling ("none","!win32", bare strings) since bun writes those forms back intobun.lock.Getting the full opencode repro from #77 green surfaced one more offline killer fixed here: with
minimumReleaseAgeconfigured (opencode's bunfig sets it), bun silently rejects any cached manifest that doesn't claim extended data — it demotes the entry to expired and refetches. Synthesized manifests now sethas_extended_manifest; their zeroed publish timestamps read as "published at epoch", which passes every age gate, the right answer for lockfile-pinned versions. (Two hook-level killers found in the same investigation — lockfile drift forcing re-resolution, and thecatalog:rewrite mangling non-registry specs — apply to master independently of the manifest work and live in #102.)Testing: two new flake checks install fixtures with
bun.lockexcluded from the source, which forces bun through the manifest-cache path — without the synthesized manifests they fail with exactly the error from #71. One covers the default registry, the other uses a real alternate registry (registry.npmmirror.com). On top of that there are golden tests pinning the wyhash values and header layout, round-trip tests for the binary format, and a tripwire that fails if bun ever bumps the manifest format version, with notes on how to re-port. As an end-to-end test, the full opencode workspace — ~3,200 packages: workspaces, 330 catalog references, 15 patched deps, pre-release pins, a github dep, a pkg.pr.new tarball, vendored tarballs — installs completely offline from a cache generated by this branch, once its lockfile is refreshed to clear the drift caught by #102 (as committed upstream it fails with that PR's diagnostic instead).Compatibility: existing
bun.nixfiles build unchanged (no manifest attrs → empty manifest cache → old behavior); you get the fix by regenerating. The wasm export grew two optional trailing args for the config file contents. The obvious risk is the coupling to bun's manifest ABI, but it's been stable at v0.0.7 across 1.3.x, and the tripwire turns a future bump into a loud test failure instead of silently broken installs.Not covered here: auth for private-registry tarball fetching (the cache keys are already right, since credentials aren't part of the hashed URL, but authenticating the FODs is a separate problem), and there's no end-to-end fixture for a path-style registry URL — that's unit-tested only, since I couldn't find a public registry shaped like that to commit a fixture against.